{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# MNIST Image Classification with TensorFlow\n",
    "\n",
    "This notebook demonstrates how to implement a simple linear image models on MNIST using Estimator.\n",
    "<hr/>\n",
    "This <a href=\"mnist_models.ipynb\">companion notebook</a> extends the basic harness of this notebook to a variety of models including DNN, CNN, dropout, pooling etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import shutil\n",
    "import os\n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Exploring the data\n",
    "\n",
    "Let's download MNIST data and examine the shape. We will need these numbers ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "HEIGHT = 28\n",
    "WIDTH = 28\n",
    "NCLASSES = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get mnist data\n",
    "mnist = tf.keras.datasets.mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "# Scale our features between 0 and 1\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0 \n",
    "\n",
    "# Convert labels to categorical one-hot encoding\n",
    "y_train = tf.keras.utils.to_categorical(y = y_train, num_classes = NCLASSES)\n",
    "y_test = tf.keras.utils.to_categorical(y = y_test, num_classes = NCLASSES)\n",
    "\n",
    "print(\"x_train.shape = {}\".format(x_train.shape))\n",
    "print(\"y_train.shape = {}\".format(y_train.shape))\n",
    "print(\"x_test.shape = {}\".format(x_test.shape))\n",
    "print(\"y_test.shape = {}\".format(y_test.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "IMGNO = 12\n",
    "plt.imshow(x_test[IMGNO].reshape(HEIGHT, WIDTH));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Define the model.\n",
    "Let's start with a very simple linear classifier. All our models will have this basic interface -- they will take an image and return probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Build Keras Model Using Keras Sequential API\n",
    "def linear_model():\n",
    "    model = tf.keras.models.Sequential()\n",
    "    # TODO: Create linear model that takes the input features and outputs the normalized probabilities\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Write Input Functions\n",
    "\n",
    "As usual, we need to specify input functions for training, evaluation, and predicition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Create training input function\n",
    "train_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x = {\"image\": x_train},\n",
    "    y = y_train,\n",
    "    batch_size = 100,\n",
    "    num_epochs = None,\n",
    "    shuffle = True,\n",
    "    queue_capacity = 5000\n",
    "  )\n",
    "\n",
    "# Create evaluation input function\n",
    "eval_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    # TODO: Create eval input function. Hint: Be sure to change each of the parameters since training and evaluation are different\n",
    "  )\n",
    "\n",
    "# Create serving input function for inference\n",
    "def serving_input_fn():\n",
    "    placeholders = {\"image\": tf.placeholder(dtype = tf.float32, shape = [None, HEIGHT, WIDTH])}\n",
    "    features = placeholders # as-is\n",
    "    return tf.estimator.export.ServingInputReceiver(features = features, receiver_tensors = placeholders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Create train_and_evaluate function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " tf.estimator.train_and_evaluate does distributed training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Create train_and_evaluate function\n",
    "def train_and_evaluate(output_dir, hparams):\n",
    "    # Build Keras model\n",
    "    model = linear_model()\n",
    "        \n",
    "    # Compile Keras model with optimizer, loss function, and eval metrics\n",
    "    model.compile(\n",
    "        optimizer = \"adam\",\n",
    "        loss = \"categorical_crossentropy\",\n",
    "        metrics = [\"accuracy\"])\n",
    "        \n",
    "    # Convert Keras model to an Estimator\n",
    "    estimator = tf.keras.estimator.model_to_estimator(\n",
    "        keras_model = model, \n",
    "        model_dir = output_dir)\n",
    "\n",
    "    # Set estimator's train_spec to use train_input_fn and train for so many steps\n",
    "    train_spec = tf.estimator.TrainSpec(\n",
    "        input_fn = train_input_fn,\n",
    "        max_steps = hparams[\"train_steps\"])\n",
    "\n",
    "    # Create exporter that uses serving_input_fn to create saved_model for serving\n",
    "    exporter = tf.estimator.LatestExporter(\n",
    "        name = \"exporter\", \n",
    "        serving_input_receiver_fn = serving_input_fn)\n",
    "\n",
    "    # Set estimator's eval_spec to use eval_input_fn and export saved_model\n",
    "    eval_spec = tf.estimator.EvalSpec(\n",
    "        input_fn = eval_input_fn,\n",
    "        steps = None,\n",
    "        exporters = exporter)\n",
    "\n",
    "    # Run train_and_evaluate loop\n",
    "    tf.estimator.train_and_evaluate(\n",
    "        estimator = estimator, \n",
    "        train_spec = train_spec, \n",
    "        eval_spec = eval_spec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This is the main() function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "OUTDIR = \"mnist/learned\"\n",
    "shutil.rmtree(OUTDIR, ignore_errors = True) # start fresh each time\n",
    "\n",
    "hparams = {\"train_steps\": 1000, \"learning_rate\": 0.01}\n",
    "train_and_evaluate(OUTDIR, hparams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I got:\n",
    "\n",
    "`Saving dict for global step 1000: categorical_accuracy = 0.9112, global_step = 1000, loss = 0.32516304`\n",
    "\n",
    "In other words, we achieved 91.12% accuracy with the simple linear model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<pre>\n",
    "# Copyright 2017 Google Inc. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#      http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "</pre>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
